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Abstract: The limit behavior of inductive logic programs has not been explored, but 
when considering incremental or online inductive learning algorithms which usually 
run ongoingly, such behavior of the programs should be taken into account. An 
example is given to show that some inductive learning algorithm may not be correct 
in the long run if the limit behavior is not considered. An inductive logic program is 
convergent if given an increasing sequence of example sets, the program produces a 
corresponding sequence of the Horn logic programs which has the set-theoretic limit, 
and is limit- correct if the limit of the produced sequence of the Horn logic programs 
is correct with respect to the limit of the sequence of the example sets. It is shown 
that the GOLEM system is not limit-correct. Finally, a limit-correct inductive logic 
system, called the prioritized GOLEM system, is proposed as a solution. 
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1. Introduction 

As information increases exponentially, it becomes more and more important to discover 
useful knowledge in massive information. Inductive logic programming is used in learning a 
general theory from given examples. In incremental learning, the examples are usually given 
one by one. After a new example is obtained, the current theory learned from previous 
examples might need to be updated to fit all the examples given so far. Thus we get a 
sequence of theories Hi, II2, • • • , n„, • • •. Sometimes there might be infinitely many examples 
and this procedure may not stop, i.e., there may not exist a natural number k such that 
Life = Llfe + i = • • •. For example, if we restrict the theories to Horn logic programs, then 
there exists some Herbrand interpretation / such that we will never find a finite program 
LI whose least Herbrand model is equal to J, because the set of Herbrand interpretations 
is uncountable while the set of finite Horn logic programs is only countable ([10]). So we 
should consider some kind of the limits of theories which should be correct with respect to 
all the examples. 

Formally introducing the set-theoretic limits of sequences of first order theories into logic 
and computer science, and using theory versions as approximations of some formal theories 
in convergent infinite computations are the independent contributions by Li. Li ([4], [5]) 
defined the set-theoretic limits of first order theories, and thereon gave a formal system of 
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the inductive logic. Precisely, given a sequence {II n } of the first order theories, the set- 
theoretic limit of {n„}, denoted by II = lim IT n , is the set of the sentences such that every 

n^oo 

sentence in IT is in almost every IT n , and every sentence in infinitely many II„'s is in II also. 
The set-theoretic limit does not always exist for any sequence of the first order theories. In 
the following sections we use the limits as the set-theoretic limits. 

The limit behaviors of an inductive logic program should be an important research topic 
which has not been explored. Nowadays most of the softwares and algorithms are incremental 
or online, which run in the long run. When we consider the correctness problem of such 
softwares and algorithms, their limit behaviors should be taken into account. In this paper 
we focus on incremental inductive learning algorithms in inductive logic programs. Assume 
that examples come in sequences, let E n be the example set at time n. An inductive learning 
algorithm A produces a theory A(E n ), for every n, which is correct with respect to E n . 
Later we shall give an example to show that some inductive learning algorithms may not be 
correct in the long run if the limit behavior is not considered. Concerning the limit behavior, 
a reasonable inductive learning algorithm A should satisfy the following conditions: 

• The convergence: Given a sequence {E n } of the example sets such that E\ C E2 C 
E3 C • • • , {A(E n )} has the set-theoretic limit; 

• The limit-correctness: The limit of {A(E n )} should be correct with respect to the limit 
of {E n }, that is, for any e e lim^oo E n , lim^oo A(E n ) h e; 

Based on the above requirements, we consider the empirical ILP systems: FOIL, GOLEM 
and MOBAL. Because FOIL and MOBAL are function-free, we focus on the GOLEM system. 

In the following, our discussion is based on a fixed logical language which contains only 
finitely many predicate symbols. 

The authors ([7]) considered the limit behavior of the Horn logic programs, and proved 
the following theorem: 

Theorem 1.1 ([7]). Given a sequence {n„} of Horn logic programs, if II = lim U n 

n^oo 

exists and for every sufficiently large n, II n satisfies an assumption that for every clause in 
U n , every term occurring in the body also occurs in the head, then 

lim M(U n ) =M{ lim n„), 

n — >oo n — >oo 

where M is an operator such that for any Horn logic program n, M (U) is the least Herbrand 
model of n. 

To consider the limit-correctness of inductive logic programs, we assume that the induc- 
tive logic programs satisfy the convergence. Given an inductive logic program A, if for every 
positive example set E n , A{E n ) is a Horn logic program which is correct with respect to 
E n , i.e., M{A{E n )) D E n , and A(E n ) satisfies the assumption, then by theorem 1.1, 

M{ lim A{E n )) = lim M{A{E n )) D lim E n , 

n — >oo n — >oo n — >oo 

so A is limit-correct. Therefore, to make A satisfy the limit-correctness, we should design 
A such that for any input E, A(E) is a Horn logic program satisfying the assumption. We 
shall give two examples to show that the current GOLEM system may not always produce a 
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logic program satisfying the assumption. Wc modify the GOLEM system to be a prioritized 
one in which a priority order is defined on the literals. In detail, let Q and V be the GOLEM 
algorithm and the prioritized GOLEM algorithm such that for any example set E, Q{E) 
and V{E) are the Horn logic program produced by the GOLEM system and the prioritized 
GOLEM system, respectively. Then, for a set E of examples, Q(E) may not satisfy the 
assumption, but V(E) satisfies the assumption. Hence, V is limit-correct. 

The paper is organized as follows. In section 2, we shall give the basic definition of the 
GOLEM system, the distance on terms and formulas, and the set-theoretic limits. In section 
3, we shall consider the limit-correctness of the GOLEM system, and give two examples to 
show that the GOLEM system is not limit-correct and sensitive to the ordering of the 
examples. In section 4, we shall propose the prioritized GOLEM system, and prove that 
the prioritized GOLEM system is limit-correct, and not sensitive to the ordering of the 
examples. The last section concludes the paper. 

Our notation is standard, references are [1,2,6]. 

2. The GOLEM system 

In this section, we first give a basic introduction to the GOLEM system. Similar to 
Nienhuys-Cheng's definition of the distance on terms and formulas ([10]), we give the follow- 
ing definition. 

Definition 2.1. Let / and g be an n-ary and an m-ary function symbols, respectively. 
The distance p is defined as follows. 

(2.1) p(t, t) = 0, for any term t; 

(2.2) Iff ^g, then p{f{t 1 ,...,t n ),g(s 1 ,...,s m )) = 1; 

f>i\ ftu + \ ti w max{p(t t ,s t ) | 1 < i < n} 

2.3) p(f(t u ...,*„), /(si, s„ = — . 

m&x{p(ti 7 Si) 1 < i < n\ + 1 

where t\, ...,t n , s\, s m are terms. 

The distance defined above is a little different from the one given by Nienhuys-Cheng in 

that the value which the distance can take has a simple form — for some natural number. 

m 

Such a distance is used to measure the difference between two trees in graph theory. Every 
term t can be taken as a tree T t . For example, t — f(t\, ...,i„), the tree T t has a root with 
symbol / and n-many children T tl , ...,T tn . We say that two terms t and t' are the same to 
depth m if T t and T t < are the same to depth m. 

Given two clauses C\ , C 2 , to compute the least general generalization of C\ and C 2 , 
denoted by lgg(Ci, C 2 ), we give the following procedure: 

Step 1. Given two terms t = f(h, ...,t n ) and s = g(si, s m ), 



lgg(i, s) = 
where v is any new variable. 



v if / ^ g 

/(lgg(ii,si),...,lgg(*n,s n )) if f = g, 
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Step 2. Given two literals h — (-i) fcl p(ii, ...,t n ) and l 2 = (^) k ' 2 q(si, s m ) 
lgg(*i,fc) = 



undefined if k\ ^ k 2 or p 7^ q 

(-.) fcl pi(lgg(ti,si), ...,lgg(i n ,s n )) if fci = k 2 and p = q, 



where fci = or 1, (-.)° = , (-.)* = -.. 

Step 3. Given two clauses C\ = {h, ...,l n } and C 2 — l' m }, 

Igg(Ci, C 2 ) = {lggft, : 1 < % < n, 1 < j < m, \gg(h, is defined}. 

Given two sets A, B of clauses, define 

lgg(A B) - {lgg(C l5 C 2 ) : C\eA,C 2 e B, d(C\, C 2 ) = mm{d(C u C 2 ) : C 2 e B} 

& lgg(Ci,C 2 ) exists}. 

Mugglcton and Feng [8] showed that if T is a finite set of ground literals then the rlgg of 
two clauses of C\ and C 2 with respect to IT is the lgg of II — > C\ and II — > C 2 . Based on 
the property, they gave the ILP learning system GOLEM which is the only learning system 
explicitly based on the notion of relative least general generalization. 
The GOLEM system: 

Suppose we are given a logic program T (i.e., a background knowledge) and two examples 
(two ground atoms) E\ and E 2 such that V \f E\ and Y \f E 2 . We construct the lgg C of Ex 
and E 2 relative to T, so T A C h E\ A E 2 and C is used only once in the derivation of both 
Ei and E 2 . Let T = {p x , ...,p„, ...}. 

Define Ci = ((^pi V -.p 2 V • • •) V £1), C 2 = ((^Pi V -.p 2 V • • •) V E 2 ), set C = lgg(Ci,C 2 ). 

Definition 2.2. Given a sequence {A n } of sets of formulas, the set-theoretic limit of 
{A n } exists, denoted by lim^oo A n , if 

lim^ooAn = Ihn^^An 

where 

liuin^ooAn = {ip : 3°°n(^ G A n )}; 
Mkn^oo A n = W ■ 3n Vn > n (ip G A„)}, 
where 3°°n means that there are infinitely many n. 

3. The limit-correctness of the GOLEM system 

In this section we consider the limit-correctness of the GOLEM system, and give two 
examples to show that the GOLEM system is not limit-correct, and sensitive to the ordering 
of the examples. 

Let p be a predicate saying that x is an even number if p(x); and s be the successor 
function, i.e., s(x) is the successor of x. Let 

i?„-{p(0),p( S 2 (0)),...,p( S 2 "(0))}. 
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There are two inductive learning algorithms to produce two different theories for E n . 

Case 1. One inductive learning algorithm produces T n = {p(0);p(s 2 (x)) <— p(x)}, given 
E n . Then T n is a Horn logic program and the least Herbrand model of T n is 

M n = {p(0),p( s 2 (0)),...,p( s 2m (0)),...}. 

Then we have that 

T = lim^oo T n =Ti; 
M = lim IWOO M„ = Mi. 

The least Herbrand model of T is M. 

Case 2. Another inductive learning algorithm produces S n — {p(s 2n (0));p(x) <— 
p(s 2 (x))}, given E n . Then, S n is a Horn logic program and the least Herbrand model, 
say N n of S n is E n . But 

S = limn^oo S n = {p(x) <- p(s 2 (x))}; 
N = ]im n -> 00 iV n = limn^oo = Mi. 

The least Herbrand model, say M(S), of S is equal to the empty set. Then 

M(S) + N. 

That is, for any e E N, S \f e. 

Example 3.1. Assume that E n is given as above, then the GOLEM system produces 
the following sequence of the Horn logic programs: 

To -MO)}; 

Ti = {p(0);7n}; 

T 2 ={p(0);7ri;lgg(7ri,7r 2 )} 

= { P (0);tt 1 ,p(s 2 (x)) ^ P (x)} 

= { P (0):p(s 2 (x))^p{x)}, 
T 3 =T 2 , 

=T 2 , 



where 

^2 ={-W0)V P ( S 2 (0))),p(/(0))} 

= M0), P (s 4 (0))} a {^( S 2 (0)),p( S 4 (0))}, 
tti = {^(0),p(s 2 (0))}, 

lgg(7Tl, 7T 2 ) = { P (s 2 (x)), ->p(x)}. 
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We change the ordering of the occurrences of p(s 2n (s)) , s and see what the GOLEM 
system gets. 

Example 3.2. Assume that 

E'o ={P(* 4 (0))}, 

E[ = E U{p(s 2 (0))}, 

E' 2 =E 1 U{p(0)}, 



E> k =i?3 fc -iUM S 6fc+4 (0))}, 
E> k+1 =E 3k U{p( S ^(0))}, 
E' 3k+ 2 =£3HiU{# fc (0))}, 



Then the GOLEM system produces the following sequence of the Horn logic programs: 

50 ={P(* 4 (0))}; 

51 ={p(s 4 (0)); 7 i}; 

5 2 ={p( S 4 (0)); 7 i;lgg(7i,72)} 

= {p(s 4 (o));ii;p(x)^p(s 2 (x))} 

= {p( S H0));p(x)^p( S \x))}, 
S 3k+t = {p( S ek +^(0)):p(x) - p(s 2 (x))}, 

where 

71 ={^(s 4 (0)),p(* 2 (0))}, 

72 - {^ 4 (0)),P(0)} A {^( S 2 (0)),p(0)}, 

lgg(7i,72) = {p(x),^p(s 2 (x))}. 

By the above discussion, this example shows that the GOLEM system is not limit-correct, 
and sensitive to the ordering of the examples. 

4. The prioritized GOLEM system 

In the section, we give the prioritized GOLEM system which is limit-correct and not 
sensitive to the ordering of the examples. In [6] the authors proposed an assumption on 
the Horn logic programs which guarantees that the least Herbrand model of the limit of 
a sequence of Horn logic programs is the limit of the least Herbrand models of the logic 
programs, and the least Herbrand model of the former limit is stable with respect to the 
sequences of the Horn logic programs. 

Definition 4.1. A clause is simple if every subterm occurring in the body of the clause 
occurs in the head of the clause. A logic program is simple if every clause in it is simple. 
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The assumption in section 1 requires that the Horn logic program be simple. 
Given an example set E, we modify the GOLEM system to be a new alogrithm V such 
that V(E) is simple. 

We first define a relation -< on the literals. Given two literals I and I', we say that I has 
a higher priority than I', denoted by I ~< I', if every sub-term occurring in I occurs in /'. 
Proposition 4.2. -< is a pre-order, that is, -< is reflexive and transitive. 

The prioritized GOLEM system: 

Suppose we are given a logic program T (i.e., a background knowledge) 
and two examples (two ground atoms) E\ and _E 2 such that T \f E\ and 
r \f _E 2 . We construct the lgg C of E\ and _E 2 relative to T, so T A 
C h E1AE2 and C is used only once in the derivation of both E\ and 
E 2 . Let T = {pi, ...,p n , ...}. 

Define C x = ((->gi V -.g 2 V • • •) V ei), C 2 = ((->gi V -.g 2 V • • •) V e 2 ), set C = 
lgg(Ci,C 2 ), where qi, g 2 , e\ is the set ru{i?i} under the order -<!, that 
is, {qi, q2, ei} = T U {Ei} and e\ qi for every i; similarly define 
{q'^q'2, ...,e 2 }. 

The prioritized GOLEM system on sequences: 

Given a sequence {E n } of example sets, at stage n, input E n . If E n 7^ 
Ei for any i < n then use the GOLEM system directly to produce a Horn 
logic program, say V(E n ); otherwise, find the least i < n such that 
E n -< E i+ i, then use the GOLEM sytem to Uj>i Ej with background knowledge 
V{Ei). Let V(E n ) be the resulted theory. 

Theorem 4.3. Given an example set E, V(E) is simple. 

Proof. By the definition of the pre-order, when a clause n is enumerated in V(E) the 
instance of the head of tt always has the lowest priority among E. This guarantees that ir is 
simple. 

Therefore, we have the following theorem. 

Theorem 4.4. V is convergent, and limit-correct. That is, given an increasing sequence 
{E n } of positive example sets, 

(4.1) {V{E n )} is a sequence of Horn logic programs which has the set-theoretic limit; 

(4.2) linin^oo V{En) is limit-correct with respect to linin^oc E n . 

Proof. By the discussion in section 1, we only need to prove (4.1). To prove (4.1), by the 
definition of the pre-order -<, -< is well-founded. Given an example e, there are only finitely 
many e' with e' -< e. Assume that when e is enumerated in E n for some n, a clause 7r is 
produced. 7r is extracted out of ViEn) only when an example e' -< e is enumerated in E n > 
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for some n' > n. Hence, tt cannot be enumerated in V(E n ) and be extracted out of V(E n ) 
for infinitely many n. Therefore, {V(E n )} has the set-theoretic limit. 

5. Conclusion and further work 

When input a set E of examples, the prioritized GOLEM system produces a Horn logic 
program V(E) which is simple. The simple Horn logic programs have many good properties 
that a common Horn logic program has not. The prioritized GOLEM system is based on 
the syntactical properties of the examples, that is, the priority order -< on the literals, which 
make the prioritized GOLEM system useful in diverse applications. 

A further work could be based on the distance defined on the terms or formulas of 
a logical language. Such the distance definitions can be the ones given by Fitting([3]) or 
Nienhuys-Cheng([9]). Then the Cauchy sequences of terms or formulas can be defined. Then 
the convergence and the limit-correctness can be defined in terms of the Cauchy sequences 
of the example sets and the Horn logic programs. It is conjectured that the prioritized 
GOLEM system also satisfies the convergence and the limit-correctness defined on Cauchy 
sequences. 



References: 

[1] F. Bergadano and D. Gunetti, Inductive Logic Programming: from machine learning to 
software engineering, The MIT Press, Cambridge, Massachusetts, London, 1996. 

[2] M. Dahr, Deductive Databases: Theory and Applications, International Thomson Com- 
puter Press, 1997. 

[3] M. Fitting, Metric methods, three examples and a theorem, J. of Logic Programming 
21(1994), 113-127. 

[4] W. Li, An Open Logic System, Science in China (Scientia Sinica) (series A), 10(1992) (in 
Chinese), 1103-1113. 

[5] W. Li, A logical Framework for Inductive Inference and Its rationality, in Fu,N.(eds.): 
Advanced Topics in Artificial Intelligence, LNAI 1747, Springer, 1999. 

[6] J. W. Lloyd, Foundations of Logic Programming, Springer- Verlag, Berlin, 1987. 

[7] S. Ma, Y. Sui and K. Xu, The limits of the Horn logic programs. Proc. of 18th 
International Conference on Logic Programming (poster session), Denmark, to appear, 



2002. Full paper is available at http:/ /www.nlsde.buaa.edu.cn/^kexu 



[8] S. Muggleton and C. Feng, Efficient inductive of logic programs, in: Proc. of the First 
Conf. on Algorithmic Learning Theory, Tokyo, 1990, Ohmsha. 

[9] S. H. Nienhuys-Cheng, Distances and limits on Herbrand Interpretations, Proc. of the 
8th International Workshop on Inductive Programming, LNAI 1446, Springer, 1998, 
250-260. 



8 



[10] S. H. Nicnhuys-Chcng, Distance between Herbrand Interpretations: A Measure for 
Approximations to a Target Concept, in: Proc. of the 7th International Workshop on 
Inductive Logic Programming, LNAI 1297, Springer, 1997, 213-226. 

[11] G. Plotkin, A note on inductive generalization, in B. Meltzer and D. Michies, eds., 
Machine Intelligence 5, 153-163, Edinburgh Univ. Press, 1970. 

[12] G. Plotkin, A further note on inductive generalization, in B. Meltzer and D. Michies, 
eds., Machine Intelligence 6, 101-124, Edinburgh Univ. Press, 1971. 



9 



